The present invention relates to a method and apparatus for computing projections of a two-dimensional image, obtained using an image sensor as a whole, in a plurality of directions and estimating the direction of movement from a result of the computing, and to a computer-readable storage medium containing a program for the method.
Description is made hereinafter for a direction instructing apparatus based on the conventional technology. For instance, in information equipment such as a personal computer, a pointer displayed on a screen is moved according to the movement of a mouse or the like used as a pointing device. Namely, a pointer displayed on a screen is moved according to, for instance, the direction of movement or the speed of the pointer computed from the coordinates (position) on a screen indicated by the pointer or the like as well as from rotation of a ball incorporated therein.
With the pointing device as described above, use thereof is limited to a space on a desk, namely it is necessary to secure an operational space, and the pointing device can not be operated in a small space such as that in, for instance, a running car. Even when the required space is available, if the space does not have a surface that gives a proper friction or if it has a vertical face (such as a wall surface), the ball in the mouse cannot rotate and the pointer on a screen cannot stably be operated.
To solve the problem as described above, it is conceivable to provide, for instance, an image sensor in a pointing device in order to move in instructing position (a pointer) on a screen to a particular position by computing the direction of movement of the image sensor itself from the movement of an image obtained by the image sensor according to a result of computing. With the method described above, it is possible to stably operate a pointer on a screen even at a surface which does not provide proper friction or on a vertical surface as if the pointer were operated on a desk.
To realize the pointing device as described above, it is necessary to estimate the movement of an image obtained by the image sensor as well as to move the pointer by updating a mouse position or a position (coordinates) of the pointer on the screen according to a result of estimation. As a technical document enabling the processing as described above, there is, for instance, Japanese Patent Laid-Open Publication No. HEI 9-134250 disclosing cursor position control on a video display unit. In the method disclosed in this document, a movement vector included in an image comprising 32xc3x9732 pixels is computed by means of correlational operation.
Generally, the resolution movement within one pixel can not be detected, therefore, in order to account for a relatively slow movement which can not be detected at this resolution, it is necessary to raise the resolution of the image itself. Thus, in order to respond to the relatively slow movement as described above in the positional control as disclosed in the technical document above, it is necessary to increase the number of calculations required for a correlational operation for a movement vector and memory capacity of an image buffer according to a resolution shown by Nxc3x97N pixels.
As described above, with the conventional technology, a cursor or the like on a screen can be controlled by computing a movement vector included in an image consisting of Nxc3x97N pixels according to an image picked up with an image sensor by means of a correlational operation.
With the conventional technology as described above, however, the number of calculations required for correlational operations for a movement vector increases according to an improvement in the resolution of the image sensor, so that a more expensive CPU is required for high speed operation. Further, capacity of the memory which stores the image data and a result of computing also increases according to an improvement in the resolution of the image sensor, so that a memory with a larger capacity is required. Therefore, with the conventional technology, there occurs the problem that low cost packaging of a pointing device is difficult.
It is an object of the present invention to obtain a direction instruction apparatus which does not require a special operation space and can operate a pointer on a screen, even on a surface which does not provide proper friction or a vertical surface, and also which enables low cost packaging using a low price CPU and a small capacity memory, a direction estimating method applicable for the apparatus, and a computer-readable storage medium containing a program for the method.
With the present invention, from two-dimensional image data obtained with an image sensor, projections of the two-dimensional image as a whole in a plurality of directions are computed, and a one-dimensional movement vector for each projection is computed from the computed projections. Then a two-dimensional movement vector for the two-dimensional image as a whole is estimated from the plurality of one-dimensional movement vectors. As described above, with the direction instructing apparatus, a two-dimensional movement vector is not directly computed from pixel data for a two-dimensional image like in the conventional technology, but at first, projections for a two-dimensional image are computed, and then the two-dimensional movement vector is estimated. More specifically, a two-dimensional movement vector is not computed directly from, for instance, 32xc3x9732 (=1024) pixel data, but at first 32 rows+32 columns (=64) projections are computed, and then the two-dimensional movement vector is estimated.
Therefore, in the direction instructing apparatus according to the present invention, a number of data can substantially be reduced and in association with this number of calculations performed by the CPU and a memory capacity for storing data can largely be reduced. In other words, a cheaper CPU and memory can be packaged and a function of a pointing device can be realized with a low cost system. Further, as number of calculations performed by the CPU and a memory capacity for storing the data can largely be reduced, the computing speed can be increased. In addition, where the direction instructing apparatus according to the present invention is used as a mouse, this mouse operates based on recognition of movement by means of image processing, so that it is not necessary to secure any special operating space, and further it is possible to stably operate a pointer on a screen even at a surface which does not provide proper friction or even at a vertical surface.
With the invention, for instance, when a pointer is displayed at a position on a screen, positional information represented by coordinates (positional information for a pointer being currently displayed) is updated according to a two-dimensional movement vector estimated by a direction estimating unit, and the position of the pointer on a screen is moved to the updated coordinate position according to positional information represented by the updated coordinates as well as to a size of the vector (speed of movement). With this configuration, the direction instructing apparatus according to the present invention can realize a mouse which does not require any specific operating space, and further a pointing device pointing a point on a screen used in a conference or during a lecture (corresponding to a conventional type of pen-light or stick) or a pointing device for game operation (used for operation in a shooting game or a game in which a characters are moved) can be realized by using the two-dimensional image and also by utilizing the characteristics that a pointer or the like displayed on the screen can be moved.
With the present invention, from the data for a two-dimensional image obtained with an image sensor, projections for the two-dimensional image as a whole in a plurality of directions are automatically outputted, and a one-dimensional movement vector for each projection is computed from the projections. Then, a two-dimensional movement vector for the two-dimensional image as a whole is estimated according to the plurality of one-dimensional movement vectors. As described above, with the direction instructing apparatus according to the present invention, as described above, a two-dimensional movement vector is not directly computed from pixel data of the two-dimensional image, but at first projections for the two-dimensional image are obtained, and then the two-dimensional movement vector is estimated. To explain more specifically, a two-dimensional movement vector is not computed directly from, for instance, 32xc3x9732 (=1024) pixel data, but at first projections for 32 rows+32 columns (=64) are obtained, and then the two-dimensional movement vector is estimated.
Because of this configuration, with the direction instructing apparatus according to the present invention, a number of data can largely be reduced and in association number of calculations performed by the CPU and a memory capacity for storing the data can substantially be reduced. In other words, a cheaper CPU and memory can be packaged and a function of a pointing device can be realized with a lower cost system. Further as the number of calculations performed by the CPU and a memory capacity for storing the data can substantially be reduced, the computing speed can be increased. In addition, when the direction instructing apparatus according to the present invention is used as a mouse, this mouse operates based on recognition of movement by means of image processing, so that, different from a mechanical mouse with a track ball used in the conventional technology, it is not necessary to secure any specific operating space, and further it is possible to stably operate a pointer on a screen even at a surface which does not provide proper fraction or on a vertical surface.
With the invention, it is not necessary to compute with a movement computing unit a difference between a projection output at a current point of time and that at a previous point of time. Therefore, in this direction instructing apparatus according to the present invention, number of calculations by a CPU can further substantially be reduced, and operations can speedily be realized.
With the present invention, for instance, an artificial retina LSI operating as a semiconductor integrated circuit automatically outputs, from data for a two-dimensional image obtained with an image sensor, at least either one of projections for the two-dimensional image as a whole in a plurality of directions or a difference between a projection output at a current point of time and that at a previous point of time. Therefore, in the direction instructing apparatus according to the present invention, number of calculations performed by the CPU can further substantially be reduced, and also a total volume of required software can substantially be reduced.
With the present invention, when a pointer is displayed at a position on a screen, positional information represented by coordinates (positional information for a pointer being currently displayed) is updated according to a two-dimensional movement vector estimated by a direction estimating unit, and the position of the pointer on a screen is moved to the updated coordinate position according to positional information represented by the updated coordinates as well as to a size of the vector (speed of movement). Therefore, the direction instructing apparatus according to the present invention can realize a mouse which does not require a specific operating space, and further realize a pointing device pointing a position on a screen in a conference or during a lecture (corresponding to a peen-light or stick based on the conventional technology) or a pointing device for game operations (operations in a shooting game or a game in which characters are moved) by making use of the capability of moving a pointer or the like displayed on a screen.
With the present invention, at first, projections for a two-dimensional image in a plurality of directions at a current point of time are stored, and then a difference between a projection output at this current point of time and that at a previous point of time previously stored is computed, a one-dimensional movement vector for each projection is computed from the difference, and then these one-dimensional movement vectors are smoothed and outputted. Finally, a two-dimensional movement vector for the two-dimensional image as a whole is estimated by synthesizing the plurality of smoothed one-dimensional movement vectors.
With this method, the direction instructing apparatus according to the present invention can substantially reduce a required number of data, and in association with this makes it possible to substantially reduce number of calculations performed by the CPU as well as a memory capacity for storing the data, so that: it is possible to realize a function such as that of a pointing device with a low cost system. Further, as the number of calculations performed by the CPU and the memory capacity for storing the data can substantially be reduced, also a higher processing speed can be realized. In addition, when the direction instructing apparatus according to the present invention is used as a mouse by using this method, this mouse operates based on recognition of movement by means of image processing, so that, different from a mechanical mouse with a track ball based on the conventional technology, it is not necessary to secure a specific operating space, and further it is possible to stably operate a-pointer on a screen even at a surface which does not provide proper friction or on a vertical surface.
With the present invention, a computing algorithm for a one-dimensional movement vector can be clarified. Therefore, it becomes possible to easily and accurately compute a one-dimensional movement vector.
With the present invention, a program makes it possible to at first store projections for a two-dimensional image in a plurality of directions at a current point of time, then compute a difference between a projection output at the current point of time and that at a previous point of time, compute a one-dimensional movement vector for each projection from the difference, and finally smooth and output the one-dimensional movement vectors. Finally, the program makes it possible to estimate a two-dimensional movement vector for the two-dimensional image as a whole by synthesizing the plurality of smoothed one-dimensional movement vectors.
When this program is executed, as it is possible, with the direction instructing apparatus according to the present invention to substantially reduce a number of data and in association with this substantially reduce the number of calculations performed by the CPU as well as a memory capacity for storing data. In other words, it is possible to package therein a cheaper CPU and memory, and a function of a pointing device can be realized with a low cost system. Further, as it is possible to substantially reduce the number of calculations performed by the CPU and a memory capacity for storing the data, also high speed processing can be realized. In addition, when this program is down-loaded and the direction instructing apparatus according to the present invention is used as a mouse, this mouse operates based on recognition of movement by means of image processing, so that, different from a mechanical mouse with a track ball based on the conventional technology, it is not necessary to secure a specific operating space, and also it is possible to stably operate a pointer on a screen even at a surface which does not provide proper friction or on a vertical surface. Moreover, as the program can be down-loaded from a computer, it is possible to easily change or add a program.
With the present invention, a computing algorithm for a one-dimensional movement vector is clarified. Therefore, it becomes possible to easily and accurately compute a one-dimensional movement vector. In addition, the program can be downloaded from a computer and hence it is possible to easily change or add a program.
Other objects and features of this invention will become apparent from the following description with reference to the accompanying drawings.